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DECISION SUPPORT SYSTEM AND METHOD 
FIELD OF INVENTION 

5 In general terms, the invention relates to the field of Decision Analysis or Decision 
Support and in particular to additive points systems. 

BACKGROUND TO THE INVENTION 

10 Additive points systems (APSs), a type of multi-attribute utility or value model or 
multiple criteria decision analysis tool, are also known as * linear', 'point-count' and 
'scoring* systems. APSs are widely and increasingly used worldwide. 

APSs represent a relatively simple solution to the pervasive problem faced by decision 
15 makers with multiple criteria, attributes, or characteristics to consider, particularly when 
ranking alternatives, individuals, or candidates. Hereinafter in this document alternatives, 
individuals, or candidates are referred to generically as * alternatives', and criteria, 
attributes, and characteristics are referred to generically as 'criteria'. 

20 Specifically, APSs serve to combine the features or performance of an alternative on 
multiple criteria to produce a single ranking of alternatives with respect to an over- 
arching criterion (such as, for example, the order in which to treat patients), or, more 
simply, to reach a decision (for example, deciding whether or not to admit an immigrant 
or detennining the best site for a building). 

25 

APSs have been found in many studies to be more accurate than 'expert' decision makers 
in the respective fields to which they have been applied. In addition APSs are relatively 
simple to use. 

30 Two or more criteria that are deemed to be relevant to the overarching criterion are 
typically established. For each criterion, an alternative can usually be assigned to a 
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category (also referred to as giving an alternative a categorical rating), with each category 
(or categorical rating) scoring a particular number of points. The points are summed 
(hence additive points systems, APSs) to produce a total score for the alternative. 
Typically the number of points scored by a category will be higher if it is considered that 
5 an alternative that has that particular categorical rating for the corresponding criterion 
should generally be considered favourably with respect to the overarching criterion. 

Alternatives are ranked with respect to the over-arching criterion according to their total 
scores. Usually the higher an alternative's score the higher its ranking, and the scores 
10 typically have no other meaning than this. In some cases, alternatives are declined or 
rejected altogether from further consideration if a particular score 'threshold' is not 
reached. 

Most APSs for elective surgeries and immigration respectively (the examples referred to 
15 above), have between five and seven criteria and two to five categories on each criterion. 
Figures 1 and 2 show an example of an APS used in Canada for prioritising patients for 
hip or knee replacement surgery. 

In this example there are seven criteria, mostly based on types of pain and functional 
20 limitations. Each criterion has a number of mutually exclusive and exhaustive categories 
on which the consulted decision maker (usually a doctor) is asked to rate the patient being 
considered For example, item 2 in Figure 1, refers to the criterion of "Pain at Rest" and 
requires that the patient be assigned to one of four categories: "None", "Mild", 
"Moderate" or "Severe". 

25 

In general, if the criteria and the categories on each have been chosen for a particular 
APS, then the point values for that APS must be determined (scored or calibrated) such 
that the resulting ranking of alternatives represents the decision makers' preferences. 

30 In addition to the arbitrary assignment of point values, there are two main existing 
approaches to calibrating APSs. 

) 
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The first regresses decision makers* judgements of the relative priorities or importance of 
a sample of real or hypothetical alternative 'profiles' in terms of the criteria (where a 
profile comprises the categorical ratings of the real or hypothetical alternative on the 
5 chosen criteria) and derives point values from the regression coefficients. Usually only a 
small proportion of existing or theoretically possible profiles is surveyed because of the 
responder burden on the consulted decision makers. 

The above-mentioned decision makers' judgements are often elicited via a visual 
10 analogue scale (VAS). Item 8 of Figure 2 is such a VAS, where the decision maker is 
asked "to rate the urgency or relative priority of this patient" between "Not urgent at all" 
and "Extremely urgent (just short of an emergency)". This 'score* and the patient's 
categorical ratings in terms of criteria 1 to 7 shown in Figures 1 and 2, along with 
analogous data for other patients, may then be used to calibrate the point values for the 
15 various criteria and categories using multiple regression techniques, as explained above. 

The second existing approach to calibrating APSs uses decision makers' judgements of 
the pairwise relative importance of the APS's criteria to derive ratio scale weights. These 
weights are then applied to normalised criteria values to derive point values. An example 
20 of this type of technique is the Analytic Hierarchical Process (AHP). Other examples are 
the Simple Multiattribute Rating Technique (SMART) and SMARTER and SWING. 

Thus the first approach assumes that decision makers' judgements have interval scale 
measurement properties and the second approach assumes they have interval and ratio 
25 scale properties respectively. Both of these measurement property assumptions are 
relatively stringent and current techniques for eliciting decision makers' judgements have 
well-known biases. With respect to the first approach, for example, the validity of the 
dependent variable (the experts' judgments) and therefore the point values derived from 
the estimated coefficients, can be criticised on two main grounds. 

30 
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First, the scaling methods such as the VAS used to elicit the experts' judgments of the 
profiles' relative priorities are based on mere introspection rather than the expression of a 
choice. Second, VAS in general may not actually have the scaling measurement 
properties required for the valuations that they produce to be interpreted as relative 
5 priorities rather than just as rankings. 



It is therefore desirable to have a method of calibrating new APSs or recalibrating or 
validating extant ones that requires only ordinal measurement properties, specifically the 
positive expression of a ranking over pairs of alternatives. It would also be desirable for 
10 this method to achieve accurate results while reducing the burden on decision makers of 
ranking pairs of alternatives by minimising the number of pairs they have to rank. 
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SUMMARY OF THE INVENTION 

In broad terms, in one form the invention provides a decision support method comprising: 
for two or more pre-defined criteria, with each criterion associated with one or more pre- 
defined categories, determining a point value for each category on each criterion by the 
pairwise ranking of profile pairs wherein each profile comprises a set of two or more of 
the criteria, each criterion in the set instantiated with one of the categories for that 
criterion. 

In broad terms in another form, the invention provides a decision support system 
comprising: a pre-determined plurality of criteria stored in computer memory, each 
criterion capable of being instantiated with one or more pre-defined categories; and a 
points calibrator configured to determine appropriate points for one or more categories of 
two or more of the criteria by performing the pairwise ranking of profile pairs, each 
profile comprising a set of two or more of the criteria, each criterion in the set instantiated 
with one of the categories for that criterion. 

In broad terms in yet another form the invention provides a decision support computer 
program comprising: an initialisation component configured to receive and store data 
representing two or more criteria and one or more categories with which each criterion 
may be instantiated; and a points calibrator configured to determine point values for one 
or more categories of two or more of the criteria by performing the pairwise ranking of 
profile pairs each profile comprising a set of two or more of the criteria, each criterion in 
the set instantiated with one of the categories for that criterion. 
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BRIEF DESCRIPTION OF THE FIGURES 

Preferred forms of the decision support method, system, and computer program will now 
be described with reference to the accompanying figures in which: 

5 

Figure 1 shows a prior art means of eliciting expert judgments for the purposes of 
calibrating an APS via multiple regression-based techniques; 

Figure 2 shows a continuation of the prior art means of eliciting expert judgments for the 
10 purposes of calibrating an APS via multimedia regression-based techniques; 

Figure 3 shows a preferred configuration of hardware for carrying out the invention; 

Figure 4 shows a decision tree identifying 12 rankings of 8 profiles possible in an 
15 exemplar APS with three criteria and two categories each and allowing strict preferences 
only and no ties (for illustrative purposes only); 

Figure 5 is a flow diagram illustrating one preferred form for carrying out the invention; 

20 Figure 6 is a table showing ambiguities from six profiles (excluding those that are 
unambiguous pairwise ranked) for an exemplar APS with three criteria and two 
categories each; 

Figure 7 is a table of total and unique ambiguities for different APSs and degrees; 

25 

Figure 8 is a continuation of the table from Figure 7 of total and unique ambiguities for 
different APSs and degrees; 

Figure 9 is a flow diagram illustrating the main components of the 'efficient ambiguities 
30 generator' described in Step 1 of the method of the invention; 
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Figure 10 is a table of sufficient (but not necessary) conditions for implicitly resolving 
S^-degree ambiguities of an exemplar APS with three criteria and two categories each; 

Figure 1 1 is a flow diagram illustrating one preferred form for carrying out the invention; 

Figure 12 is a flow diagram illustrating steps for a preferred form ambiguity generator of 
the invention; 

Figure 13 is a flow diagram illustrating preferred steps for testing whether or not an 
ambiguity is resolved for the invention; 

Figure 14 is a flow diagram illustrating preferred steps for the explicit resolution of 
ambiguities for the invention; 

Figure 15 is a flow diagram illustrating preferred steps for an undo module (whereby 
explicitly resolved ambiguities are revised) of the computer program of the invention; 

Figure 16 is a flow diagram illustrating preferred steps for a further preferred aspect of 
the invention; 

Figure 17 is a flow diagram illustrating preferred steps for determining whether a 
restricted group of alternatives has been ranked; 

Figure 18 is a flow diagram illustrating the preferred steps for determining whether the 
best alternative has been identified from a restricted group of alternatives; and 

Figure 19 is a flow diagram illustrating particularly preferred steps for determining 
whether the best set of alternatives has been identified from a restricted group of 
alternatives. 
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DETAILED DESCRIPTION OF THE PREFERRED FORMS 

The invention is primarily embodied in the methodology set out below both by itself and 
as implemented through computing resources such as the preferred resources set out in 
5 Figure 3, by way of example. The invention is also embodied in the software (or 
computer program) used to implement the methodology and in any system comprising a 
combination of hardware and software used to implement the methodology. 

In its most preferred form the invention is implemented on a personal computer, 
10 workstation, or server operating under the control of appropriate operating and 
application software. 

It will however be appreciated that the invention may also be implemented on any 
computing device with sufficient memory and processing capacity. 

15 

Figure 3 shows one preferred system architecture of a personal computer, workstation, or 
server on which the invention could be implemented. The computer system 300 typically 
comprises a central processor 302, a main memory 304, for example RAM, and an 
inpul/output controller 306. The computer system 300 may also comprise peripherals 

20 such as a keyboard 308, a pointing device 310, for example a mouse, toucbpad, or 
trackball, a display or screen device 3 12, a mass storage memory 314, for example a hard 
disk, floppy disk or optical disc and an output device 316 such as a printer. The system 
300 could also include a network interface card or controller 318 and/or a modem 320, 
The individual components of the system 300 could communicate through a system bus 

25 322. 



There are two main types of APS application, which differ mainly with respect to their 
30 scale, in terms of the number of alternatives or individuals they include. The present 
invention may be used for both types of APS application. 
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The first type of APS application may be described in general terms as involving the 
ranking of large or very large numbers of alternatives, for example, 100s, 1000s or 
millions of alternatives. These alternatives represent all theoretically possible alternatives, 
as defined by the categorical ratings for the criteria by which each alternative is 
distinguished, including alternatives that may or may not yet exist and that are purely 
theoretical or hypothetical. 

APSs are widely and increasingly used worldwide for selecting immigrants. For example, 
APSs are used by the immigration systems of New Zealand, Canada, Australia, Germany 
and the United Kingdom. They are also widely and increasingly used worldwide in most 
branches of medicine for diagnosing and prioritising patients and managing waiting lists. 
For example, APSs were recently developed by the health systems of Canada and New 
Zealand for prioritising and managing access to a wide range of elective surgeries and 
other publicly-funded health cares. 

APS may also be used in a wide range of other applications, including choosing and 
appraising employees, admitting students to restricted courses, environmental 
assessments, assessing mortgage applications and predicting parole violations, business 
bankruptcies and college graduations, for example. 

The second type of APS application is similar to Ihe first one above except that instead of 
all theoretically possible alternatives, the focus is a subset of alternatives such that the 
'best' alternative is chosen from a restricted group of alternatives and/or they are ranked. 
Thus the number of alternatives to be ranked is smaller and is usually restricted to 
alternatives that are known to the decision maker. In general, this serves to reduce the 
amount of effort required of decision makers in reaching their decision. 

Examples of this type of APS application include short-listing job applicants, appraising 
investment projects, identifying desirable new product ideas or features for 
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commercialisation, choosing the best site for a building or house or model of car to buy, 
and so on. APSs may also be used as a generic project appraisal tool. 

The decision support method, system, and computer program of the present invention is 
able to score or calibrate an entire APS involving a very large number of alternatives 
including theoretical alternatives. However the decision support method, system, and 
computer program of the present invention is also able to rank a restricted group of 
alternatives and/or choose a 'best' alternative from a restricted group before an APS is 
fully calibrated. 

The method of the present invention will now be described with reference to several 
examples, beginning with an APS with just three criteria: denoted by a, b and c; and two 
categories on each: denoted by 1 and 2; such that there are six criterion-category 
variables: al, a2, bl 9 62, cl and c2. 

For example, if this were an APS for selecting immigrants, criterion a might be 
'educational qualifications', b 'wealth' and c 'language proficiency'. Category 1 might be 
generically defined as 'low' and 2 as 'high'. Real immigration APSs, however, typically 
have at least twice as many criteria and categories as this simple example. 

By definition, the values of these variables monotonically increase with the categories 
within each criterion so that the following 'inherent inequalities' hold: al > al, b2 > bl 
andc2>cl. 

Corresponding to all possible combinations of the two categories on the aba criteria, eight 
profiles, each with a total score equation, are represented by this system (where the three- 
digit numbers are symbolic representations only): 

222 = a2 4- Z>2 + c2 
221=a2 + M + cl 
212 = a2 + M + c2 
122 = al+M + c2 
112 = al+M + c2 
121=al+62 + cl 
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211 = a2 + M+cl 
lll = al + M+cl 

With respect to determining (scoring or calibrating) the points values of APSs, the 
essence of calibrating an APS such as this one is deciding the point values of the six 
variables (al, a2, bl, b2, cl and cl) such that decision makers' preferred (or 'valid') 
overall ranking of the eight profiles (equations) is realised. 

With respect to ran k ing a restricted group of alternatives or choosing the 'best' alternative 
from the group, the essence of doing so with an APS such as this one is deciding the point 
values of some or all of the variables such that decision makers' preferred overall ranking 
of the restricted group of alternatives is realised or the 'best' one is chosen before the 
APS is fully calibrated. 

For example, such a ranking may be desired of the restricted group of four alternatives 
represented by the four profiles: 

Alternative 1 : 221 = a2 + bl + cl 
Alternative 2: 122 = al + bl + c2 
Alternative 3: 212 = al + bl + c2 
Alternative* 211 = al + bl + cl 

With respect to the full set of all eight possible profiles, the internal logic of APSs 
restricts the otherwise 8! = 40,320 possible rankings (or all possible permutations), given 
strict preferences only and no ties (for illustrative purposes only), to the 12 rankings 
represented via the decision tree shown in Figure 4. The decision tree highlights the 
inherent contingencies in the derivations of the profiles' rankings. 

Any of the 12 ran k in g s shown in Figure 4 can be produced from the six variables, 
dq>ending on the values chosen for them. Ranking #1, for example, is given by al = 0, 
al = 4, bl = 0, bl = 2, cl = 0 and cl = 1, with the total scores: 222 = 7, 221 =6,212 = 5, 
211=4, 122 = 3, 121 =2, 112= 1 and 111 =0. Alternatively, ranking #12, for example, 
is given by al = 0, a2 = 1, bl = 0, bl = 2, cl = 0 and cl = 4, with the total scores: 222 = 
7, 122 = 6,212 = 5, 112 = 4,221=3, 121 = 2, 211 = 1 and 111 = 0. 
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Similarly, different rankings of the restricted group of Alternatives 1 to 4 can be produced 
from the six variables, depending on the values chosen for them. All such rankings can 
also be identified in Figure 4. 

5 

In general, any particular ranking of profiles is determined by pairwise ranking the 
profiles vis-a-vis each other. For an APS with x criteria and y categories on each, and/" 

y x (v x -1) 

profiles, a maximum of K ^ pairwise rankings is possible. Thus for the exemplar 

APS with x = 3 andj = 2, and 2 3 (or eight) profiles, there is a maximum of 8(8 - l)/2 (or 
10 28) pairwise rankings. Similarly, for the restricted group of Alternatives 1 to 4, there is a 
maximum of 4(4 - l)/2 (or 6) pairwise rankings. 

The method of the invention may minimise the number of pairwise rankings that must be 
decided explicitly (via value judgements). In the case of the full set of all eight possible 
15 profiles, in this example, a minimum of two and a maximum of four pairwise rankings, 
rather than 28, is required, with the remaining pairwise rankings implicitly resolved as 
corollaries of the explicit rankings. The pairwise ranking performed by the invention may 
therefore comprise, in broad terms, three steps, as explained in turn below. The three 
main steps in the pairwise ranking are illustrated in Figure 5. 

20 

The first step in the pairwise ranking as illustrated at 510 involves identifying (or 
'generating') the 1 ambiguities' of the APS that is being fully calibrated and/or the 
ambiguities of the restricted group of alternatives that is being ranked or from which the 
'best' alternative or alternatives are chosen. 

25 

Ambiguities are formed from the total score equations of profile pairs whose pairwise 
rankings are a priori ambiguous, given the inherent inequalities (as explained above, al > 
al, b2> bl and c2 > cl in the present example). These are profile pairs in which one 
profile has a higher categorical rating on at least one criterion and a lower rating on at 



1 75363-5 



WO 2004/111906 



PCT/NZ2004/000121 



13 

least one other criterion than the other profile. In other words, neither of the profiles in 
the pair 'dominate' the other profile; or both profiles are 'undominated' by the other. 

For example, the pairwise ranking of profiles 221 and 212 — hereinafter referred to as 
"221 vs 212" and corresponding \oa2 + b2 + c\ vs a2 + bl+c2 — is ambiguous given 
a2 > al, bl > bl and c2 > cl, as noted above. In other words, 221 does not 'dominate 5 
212, and vice versa. 'Ambiguous profile pairs', or 'ambiguities' in their reduced form 
equation forms, may also be referred to as 'undominated pairs' and 'dilemmas' 
respectively. Hereinafter in this document 'undominated pairs' and 'dilemmas' are 
referred to generically as 'ambiguous profile pairs' or 'ambiguities'. 

On the other hand, many profile pairs are unambiguously ranked. For example, 222 (a2 + 
b2 + c2) is always pairwise ranked first and 111 (al + bl + cl) is always pairwise ranked 
second. For example, a2 + b2 + c2 (222) > al + b2 + c2 (122) and al + b2 + c2 (122) > 
al + bl + cl (111), and so on. In other words, 222 'dominates' 111; and 111 is 
'dominated' by 222. Other profile pairs are similarly unambiguously ranked, for example, 
al + b2 + c2 > al + bl +c2 (122 > 112), and so on. 

As described above, identifying and eliminating all possible unambiguous pairwise 
rankings serves to cull the rankings required for the possible profiles. In the illustrative 
example shown in Figure 4, the pairwise rankings required for the 8 possible profiles can 
be reduced from 8! permutations (40,320) to 48 (for illustrative purposes only, given 
strict preferences only and no ties) from which the 12 rankings in Figure 4 are finally 
arrived at as described below. 

The method of the invention identifies and excludes profile pairs that are unambiguously 
ranked and then focuses exclusively on profile pairs that are ambiguously ranked. It does 
this for all possible alternatives and/or the restricted group of alternatives. 

Some ambiguously ranked profile pairs can be 'reduced' by cancelling variables that are 
common to both profile equations. Thus a2 + b2 + cl vs a2 + bl + c2 (221 vs 212, as 
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above), for example, can be reduced to b2 + cl vs bl + c2 by cancelling a2 from the 
equations of both profiles. In effect, because al appears in both equations it has no 
bearing on the ranking of the two profiles that the equations represent. Such reduced 
forms may be referred to as 'ambiguities 5 . 

Moreover &2 + cl vs bl + c2 also corresponds to 121 vs 112, asal + 62 + cl vsal + b\ + 
c2 reduces to b2 + cl bl + c2 after cancelling the al terms from both profiles' 
equations. Thus b2 + cl vs bl +. c2 represents two ambiguously ranked profile pairs: 121 
vs 112 and 221 vs2l2. 

However, not all ambiguously ranked profile pairs are reducible in this fashion. For 
example, no variables can be cancelled from a2 + b2 + cl vs al + bl + c2 (221 vs 1 12), 
as none are common to both profiles' equations. Nonetheless, such irreducible 
ambiguously ranked profile pairs are also hereinafter referred to as ambiguities. 

Accordingly ambiguities can be classified by the number of criteria they contain, 
hereinafter referred to as the 'degree' of the ambiguity. Thus '2 nd -degree' ambiguities 
contain two criteria, for example, 62 + cl vs bl + c2, as above; and ^-degree' 
ambiguities contain three criteria, for example, al + b2 + cl vs al + bl + c2, as above; 
and so on. 

In general, the ambiguities for an APS with x criteria range from 2 nd -degree to ^-degree. 

The algorithmically simplest process for generating ambiguities is to first create all 
combinations of the y categories on the x criteria (that is, all profiles), and then pairwise 
rank them all against each other to identify pairs in which one profile has a higher 
categorical rating on at least one criterion and a lower categorical rating on at least one 

other criterion. There will always be a total of - ^ pairwise rankings. 
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As each ambiguously ranked profile pair is identified, it is reduced where possible by 
cancelling variables in both profiles' equations, and the ambiguously ranked profile pair 
is then retained only if the resulting ambiguity has not already been generated. In other 
words, replicated ambiguities are discarded. This process for generating ambiguities is 
referred to below as 'the simple method for generating ambiguities 5 . 

Accordingly the ambiguities for the exemplar APS with x = 3 and y = 2 from Figure 4 are 
set out in Figure 6. In Figure 6, rankings that are unambiguous are denoted by 'n.a.'. The 
blank elements of the matrix, except for the main diagonal, are mirror images of the ones 
reported, and shaded ambiguities are duplicates of ones reported earlier in the table. 

Thus it can be seen in Figure 6 that although there are nine ambiguities in total, only six 
are unique: 

(1) 62 + cl vsb\+c2 

(2) a2 + cl vsal + c2 

(3) a2 + M vsal + b2 

(4) a2 + b2 + cl vsal +bl+c2 

(5) a2 + M +c2v.sal+£2 + cl 

(6) al + b2 + c2vsa2 + b\+cl 

Clearly ambiguities (1) to (3) are 2 n -degree ambiguities (and are duplicated in the figure) 
and ambiguities (4) to (6) are S^-degree ambiguities. 

Similarly, the ambiguities for the restricted group of Alternatives 1 to 4 in the example 
mentioned earlier are: 

(1) Z>2 + cl vsb\+c2 

(2) a2 + c\ vsal +c2 

(3) a2 + M vsal+b2 

(6) al + b2 + c2 vs a2 + bl+ cl 

In general, the number of ambiguities of each degree for a given APS can be calculated 
from the two equations described below. 
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For a given APS with x criteria and y categories on each criterion, the total number of 
ambiguities (including replicas) of a given degree (z) is given by: 



The first term corresponds to X C 2 (the usual combinations formula); the second term is the 
powers of 2 and minus one; the third term is y C 2 raised to the z^-power or, alternatively, 
the sum of the first y~\ natural numbers (for example, 5(5-l)/2 =10=14-2 + 3 + 4) 
raised to the z^-power; and the last term is a power function. 

Of these ambiguities, the number of unique ambiguities (excluding replicas) of a given 
degree is given by: 

(x-z)\zl K J { 2 ) W 

The two equations (1) and (2) differ only by the term (x)/* : , The meaning of this and 
the other three terms (common to both equations) is discussed further below when an 
alternative process for generating ambiguities is described 

Equations (1) and (2) can be compared with the equation mentioned earlier for the total 
number of pairwise rankings (including unambiguous ones): — ^ — . 



Equation (2) is particularly useful because it reveals how many ambiguities at each 
degree must be resolved for any given APS. Figures 7 and 8 report the numbers of total 
and unique ambiguities for different APSs and degrees. As illustrated there, the number 
of ambiguities can be relatively large even for small values of x and y. For example, an 
APS with x = 6 and y = 4 (twice the values for the exemplar APS above) has a total of 
2,295,756 unique ambiguities across its five degrees (2 nd to 6 th ). 
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The simple method for generating ambiguities described above is further simplified if any 
profiles that are theoretically impossible are eliminated from the set to be pairwise ranked 
before any ambiguities are generated. 

5 For example, in the APS for hip or knee replacements referred to in Figures 1 and 2 it 
would be a contradiction for a patient to be rated as having "severe" "Pain on motion (for 
example, walking, bending)" while also being rated on another of the criteria as having 
the "Ability to walk without significant pain" for a distance of "over 5 blocks". Such a 
combination of categories on these two criteria is theoretically impossible and therefore 
10 all profiles that include it could be deleted from the list to be pairwise ranked. 

Similarly, when validating an extant APS (rather than calibrating a new one), the profiles 
to be evaluated can be determined from a 'stock take' of the alternatives ranked by the 
APS over its lifetime. Any other profiles that might realistically be expected in the future 
15 could be added. 

This process also serves to increase the efficiency of later steps of the preferred method 
of the invention described below, as in general the more profiles that are eliminated, the 
fewer ambiguities there are, and therefore the simpler the calibration exercise is. 

20 

Notwithstanding such refinements, the simple method for generating ambiguities 
described above may be computationally inefficient for fully calibrating an APS because 
profile comparisons and pairwise rankings that are ultimately unnecessary may be 
performed and replicated ambiguities discarded 

25 

For example, to generate the above-mentioned 2,295,756 ambiguities for an APS with x = 
6 and > = 4, as can be calculated from the data in Figure 8, 6,090,804 unnecessary profile 
pairwise rankings are performed and 5,094,900 ambiguities are discarded 

30 However, the simple method for generating ambiguities described above may be 
computationally efficient for generating the ambiguities for a restricted group of 
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alternatives. Therefore it is a particularly preferred process for generating ambiguities for 
a restricted group of alternatives, and it is also a particularly preferred process for doing 
this for the computer program of the invention. 

A particularly preferred process for generating ambiguities for fully calibrating an APS 
according to the invention is described below. This is also a preferred process for a 
computer program for fully calibrating an APS according to the invention. The main 
steps in the process are illustrated in Figure 9. 

The particularly preferred process for fully calibrating an APS, hereinafter referred to as 
the 'efficient ambiguities generator', is described with explicit reference to the three 
terms in equation (2) above (reproduced below) and a specific example. The example is 
that of generating the 3 rd -degree ambiguities for an APS with x = 5 and y = 3, of which 
there are 810 in total (as can be calculated from equation (2) and is reported in Figure 7). 

Although the equation applies to APSs with the same number of categories on the 
criteria, as explained later below, the process can be generalised to allow the number of 
categories to vary across criteria. For example, instead of y = 3 for all 5 criteria in the 
above-mentioned APS, as is applied later, criterion a could have two categories, b three 
categories, c four, and so on. 

x\ 

Equation (2)' s first term — — (= X C Z9 the combinations formula) — is the number 

(x-z)lz\ 

of combinations of z criteria that can be selected from x criteria 

For the above-mentioned APS with x = 5 (that is, criteria a, b, c, d and e), 10 
combinations of three criteria (z = 3) are possible: abc, abd, abe, acd, ace, ade, bed, bce 9 
bde and cde. 
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Each combination can be thought of as forming the 'base' for a group of ambiguities. For 
example, abc is the base for all S^-degree ambiguities centred on a + b + cvsa + b + c 
(hereinafter abbreviated to abc vs abc), such as al + b3 + c3 vs a3 + b2 + c2. 

Because the criteria in each base (10 of them in the present example) have the same 
number of categories each, the bases can be treated identically with respect to the 
following operations that correspond to the other two terms in equation (2). 

The second term (2~ -1 -l) can be interpreted as representing the number of underlying 
structures for a given degree (z). Structures are generated by first listing the numbers 
between 1 and 2 2 ' 1 - 1 in binary form using z bits, where each bit represents a criterion of 
two categories represented by either 0 or 1, 

Continuing with the example of generating the 3 ri -degree ambiguities for an APS with x 
= 5, the structures for z = 3 are: 

Oil 
010 
001 

These structures correspond to 3, 2, and 1 in binary form. For now, each structure can be 

thought of as being analogous to a profile with y = 2 for all criteria Therefore only one 

ambiguity can be created from each, which is done by finding each structure's 'Is 

complement', that is, by 'flipping the bits'; thus in the example: 

011 vs 100 
010 vs 101 
001 vs 110 

Each term (either a '0' or a T) on either side of the ambiguity structure represents a 
criterion, and for each criterion their relative magnitudes ('0' versus *V) represents the 
relative magnitude of the categories they represent (in other words 'low' or 'higji'), such 
that each structure has an underlying pattern. 
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Thus Oil vs 100 in the example above (and analogously for 010 vs 101 and 001 vs 110) 

signifies that the first of the three criteria represented (corresponding to 0 vs 1 ) has 

a lower category on the left hand side (LHS) and a higher category on the right hand side 
(RHS) of each ambiguity that is derived from it. 

5 

Similarly, the second and third criteria (corresponding to _1 1 vs _00) both have higher 
categories on the LHS and lower categories on the RHS. 



The equation's third term 




is the number of ambiguities that can be 



10 generated from each ambiguity structure, as determined by the number of categories (y) 
on the criteria in a given base. There are three steps to generating these ambiguities, as 
follows. 

First, all of the bases are matched with all of the structures. In the present example, the 10 

15 bases (abc, abd, abe, acd, ace, ade, bed, bee, bde and cde) are matched with the three 

structures (01 1 vs 100, 010 vs 101 and 001 vs 1 10) to produce 10x3 = 30 matches: 

abc vs abc with 01 1 W 100 
abc vs abc with 010 vs 101 
abc vs abc with 001 vs 1 10 
20 abd vs abd with 01 1 vs 100 

. . . and so on for another 26 matches. 

Second, the underlying 'pattern' for each match (that is, a base with a structure) is 
implemented, according to the number of categories on the bases. 

25 

For abc vs abc with 01 1 100, for example, with three categories on the criteria (y = 3), 
there are three ways each (that is, 3lgzS = 3(3 _ i)/2 ways) of representing criterion a 

with a lower category on the LHS and a higher category on the RHS (0 vs 1 ), and 

criteria b and c both with higher categories on the LHS and lower categories on the RHS 
30 (_ll™j)0): 

al vs a3 b3 vs bl c3 vs c2 
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al vs a.3 b3 vs bl c3 vs cl 

al vs al b2 vs bl c2 vs cl 

The number of combinations of categories (within the criterion) taken two-at-a-time is 
5 ~ = y ^ 2 ' ^ e out P ut uste d above embodies these combinations, where the two 

categories are simply ordered as required by the structure. 



Finally, all 3 x 3 x 3 = 27 possible combinations (in other words ~ 1 = — 

l)/2] 3 ) of these three patterns are formed, thereby obtaining all 27 S^-degree ambiguities 
1 0 corresponding to the match abc vs abc with 0 1 1 vs 1 00: 

a2 + b3 + c3 vs a3+b2 + c2 

a2 + b3 + c3 vs a3 + b2 + cl 

a2 + b3 + c2 vs a3 + b2 + cl 

a2 + b3 + c3 vs a3 + bl + c2 
15 a2 + b3 + c3vsa3+bl + cl 

a2 + b3 + c2 vs a3 + bl + cl 

a2 + b2 + c3 vs a3+bl+ c2 

a2 + b2 + c3 vs a3 +£>l+cl 

a.2 + bl + c2 vs a3 + bl + cl 
20 al + 63 + c3 vja3+*2 + c2 

al + b3 + c3 vsa3+b2 + cl 

al+b3 + c2 vs a3+b2 + cl 

al + b3 + c3vsa3+bl + c2 

al + b3 + c3vsa3+bl+cl 
25 al + b3 + c2vsa3+bl + cl 

al + b2 + c3vsa3+bl+c2 

al + b2 + c3 vs a3 + bl + cl 

al + b2 + c2 vs a3+bl + cl 

al + b3 + c3 vs a2 + b2 + c2 
30 al + b3 + c3 vs a2 + b2 + cl 

al + b3 + c2 vs a2 + b2 + cl 

al + b3 + c3 vs a2 + bl + c2 

al + b3 + c3 vs a2 + bl + cl 

al+b3 + c2 vs a2 + bl+ cl 
35 al + b2 + c3ysa2 + bl + c2 

al+b2 + c3 vs a2 + bl + cl 

al + b2 + c2vs a2 + bl + cl 
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Analogues of the three steps explained above may be performed for all matches. For each 
of the 30 matches in the present example with x = 5, y = 3 and z = 3, 27 ambiguities 
analogous to the 27 above are generated, resulting in a total of 30 x 27 = 810 (unique) 3 rd - 

degree ambiguities or., - — ^— x (2 Z_1 - 1) x \2i2Lz]l) = 10 x 3 x 27 = 810 

(x-z)\z\ \ 2 ) 



As noted earlier, the process outlined above can be generalised to allow the number of 
categories to vary across the criteria. The key difference from the process explained 
above is that the underlying 'pattern' for each match (for example, abc vs abc with 01 1 vs 
100) is idiosyncratic to the criteria included, as determined by the numbers of categories 
on each criterion. 

For example, if in the example referred to above, instead of three categories on all 
criteria, criterion a may have two categories, b may have three and c four. Thus with just 
two categories for criterion a (for 01 1 vs 100, corresponding to 0_ _ vs 1_ J, there is 

only one underlying pattern ( y(y ~^ = 2 (2 - l)/2) corresponding to a lower category on 

the LHS and a higher category on the RHS of each ambiguity that is derived: 

al vs a2 

As before, with three categories for criterion b (corresponding to _1_ vs _0_), there are 
three underlying patterns ( ^-1) = 3(3 _ 1)/2) correspondmg to a higher categoiy on 

the LHS and a lower category on the RHS: 

b3 vs b2 
b3 vs bl 
b2 vs bl 

Finally, with four categories for criterion c (corresponding to 1 vs 0), mere are six 

underlying patterns ( ^—^ = 4(4 - l)/2) corresponding to a higher category on the 
LHS and a lower category on the RHS: 

c4 vs c3 
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c4 vs c2 
c4 vs cl 
c3 vs c2 
c3 vs cl 
c2 vs cl 

By taking all 1 x 3 x 6 = 1 8 combinations of the above three sets of criteria-categories, all 
18 S^-degree ambiguities corresponding to abc vs abc with Oil vs 100 (with two, three 
and four categories respectively) may be obtained: 

al + 63 + c4 vs a2 + b2 + c3 
al + 63 + c4 vsa2 + 62 + c2 
al + b3 + c4 vs a2 + b2 + cl 
al + £3 + c3 vs a2 + b2 + c2 
a 1 + b3 + c3 a2 + 62 + cl 
al + £3 + c2 vs a2 + 62 + cl 
al + b3 + c4 vsa2 + 61 + c3 
al + 63 + c4 vsa2 + b\+c2 
al + 63 + c4 vs a2 + 61 + cl 
al + 63 + c3 vsa2 + 61 + c2 
al + 63 + c3 vs a2 + 61 + cl 
al + 63 + c2 vs a2 + 61 4- cl 
al + 62 + c4 vs a2 + bl + c3 
al +62 + c4 vsa2 + 61 + c2 
al + 62 + c4 vs a2 + bl+ cl 
al + 62 + c3 vsa2 + 61 + c2 
al + 62 + c3 vs a2 + 61 + cl 
al + 62 + c2 vs a2 4- 61 + cl 

This process is performed for all matches. But, because the numbers of categories for the 
criteria are different, each match generates an idiosyncratic set of ambiguities that 
depends on the numbers of categories on the included criteria. 

For example, the number of ambiguities generated from a6c vs a6c with 011 vs 100 (as 
above) is different to the number from abd vs abd with 01 1 vs 100 when the numbers of 
categories on criteria c and d are different. Therefore, in general the ambiguities for each 
x\ 

of the ^~y^ bases (combinations of z criteria from the x criteria) must be generated 
individually. 
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For example, in the case of the base corresponding to the particular combination of z 
criteria comprising the first z of the x criteria (criterion /, criterion 2, ... criterion z, 
where z < x), there are 2 r ~ l -l structures as before. (The choice of the first z of the x 
5 criteria, instead of any other z of the x criteria, is for notational simplicity.) Each structure 

^ [M^i)j^M&r£j x ... x ^ii(&J^j ambiguities, where^,^ ...y, ^ m e 
numbers of categories on each of these first z criteria 

The total number of z^-degree ambiguities is obtained by summing the number of 

10 ambiguities (analogous to the example) across all : bases. In general, the 

(jc -z)!z! 

number of ambiguities of each degree can be calculated from the equation described 
below. 

For a given APS with y u y 2 , ... y x categories on the respective x criteria, the equation is 
1 5 based on the following definitions. First, Y is the set of the numbers of categories on the x 
criteria: Y = {y u y 2 , . . . y x } . Second, C is the set of unordered z-tuples formed by taking all 
x \ 

possible '—— combinations of the elements of 7, z-at-a-time: C = (c I c is an 

(jc ~z)!z! 11 

unordered z-tuple from 7, as defined above}. Each of Cs elements (or sets), c u is 

numbered from 1 to x! : c u i = 1, 2, ... xl Each of c/'s elements, y ih is 
(x-z)!z! (jc-z)!z! ™ 

20 numbered from 1 to z:y v ,j = 1, 2, ... z. 

Applying these definitions, the number of unique ambiguities of a given degree (z), when 
the number of categories varies across criteria, is given by: 

25 
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Alternatively this equation can be expressed as: (2 s " 1 

For common values of y (j (yy=y\ that is, all criteria with the same numbers of categories, 
equation (3) is equivalent to equation (2) above. 

Either process explained above — for the same or, alternatively, different numbers of 
categories on the criteria — can be used to generate all of an APS's ambiguities of a 
given degree (such as 3 rf -degree, as above), or, alternatively, ambiguities can be 
generated individually. 

As each ambiguity is generated it may be tested for whether or not it is theoretically 
impossible, and therefore to be discarded or not, given the theoretically impossible 
profiles that were culled earlier (as described above). For the simple process for 
generating ambiguities described earlier, all theoretically impossible profiles are simply 
removed before ambiguities are generated from them. 

However, because the 'efficient ambiguities generator' explained immediately above 
does not generate ambiguities from profiles, the ambiguities must be tested as they are 
generated for whether or not the profile pairs that they represent have been eliminated 

For example, for an APS with* = 4 andy = 3, the reduced form al + b3 vs di + bl (i.e., 
13 vs 32 ) represents nine ambiguously ranked profile pairs: 

1311 vs 3211 

1312 vs 3212 

1313 vs 3213 

1321 vs 3221 

1322 vs 3222 

1323 vs 3223 

1331 vj3231 

1332 vs 3232 

1333 vs 3233 
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For al + b3 vs a3 + 62 to be theoretically impossible and therefore discardable, in all nine 
of these pairs at least one of the profiles — a minimum of nine and a maximum of 18 
profiles — must be theoretically impossible. If instead at least one of the nine pairs is not 
excluded, then al + b3 vs a3 + b2 is possible and therefore ought not to be discarded. 
5 Accordingly all nine pairs must be considered before it can be determined whether al + 
b3 vs a3 + b2 should be discarded or not, but as soon as one profile pair is found that has 
not been excluded, the ambiguity should be retained. 

Enumerating all such profile pairs for any given ambiguity of degree z is relatively 
1 0 straight-forward, as each profile pair is based on the ambiguity in question, augmented by 
all possible combinations of the categories on the other x - z criteria. There are therefore 
y*~ z such profile pairs when the number of categories on the criteria (y) is the same. 

As noted earlier, the termy*"* appears in equation (1) above — giving the total number 
15 of ambiguities (including replicas) of a given degree — but not in the otherwise identical 
equation (2) — giving the number of unique ambiguities (excluding replicas) of a given 
degree. 

Accordingly y?~ z can be interpreted as the number of * copies' of a particular ambiguity 
20 generated by the algorithmically simple process explained earlier. That is, in the example 
above with x = 4 and y = 3, each of the -/ " z = 3 4 " 2 = 9 pairwise profile comparisons 
generates al +b3 vs a3 + b2 (of which eight are discarded because they are replicas). 

Having generated the ambiguities, The next step of the preferred method for fully 
25 calibrating an APS (as set out at step 2 in Figure 1 1) involves explicitly resolving them, 
one-at-a-time, while identifying all other ambiguities that are implicitly resolved as 
corollaries. This step is represented in Figure 5 at 520. 

Any of the unresolved ambiguities of any degree could be selected for explicit resolution 
30 during the calibration process, however there tend to be fewer ambiguities to store at 
lower degrees, and lower degree ambiguities are easier for decision makers to resolve, 
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and so the process starts by resolving the 2 nd -degree ambiguities and then proceeds to 
resolving successively higher-degree ambiguities. If instead it were desired that the 
number of decisions be minimised at the expense of the complexity of the decisions, the 
process should instead start by resolving the highest degree ambiguities and then proceed 
to resolving successively lower-degree ambiguities. 

As ambiguities can only be resolved via value judgements, they must be decided by an 
individual 'decision maker* or group of 'decision makers', preferably with knowledge of 
the field in which the particular APS is to be applied. For example, a group of decision 
makers for a medical APS may comprise a panel of doctors and patients. Hereinafter 
decision makers (plural) are referred to. 

Accordingly the preferences of the decision makers must be probed via a series of 
questions concerning their pairwise rankings of profiles. 

As listed earlier, the ambiguities for the original exemplar APS with x = 3 and y = 2 are: 
(1) 62 + cl vs 61 + c2, (2) al + cl vs al + c2, (3) al + 61 vs al + 62, (4) al + 62 + cl vs 
al + 61+ c2, (5) al + 61 + cl vs al + 62 + cl and (6) al + 62 + cl vs al + bl+cl. 

For ambiguity (1) 62 + cl vs bl + c2, for example, the decision makers are asked, in 
essence: "Given two alternatives that are identical with respect to criterion a, which has 
the greater priority, _21 or _12?" 

If there is more than one decision maker, the process of getting answers to this and 
subsequent questions can be streamlined by asking decision makers to cast votes (perhaps 
via email) for the pairwise rankings they favour. 

However, in this type of situation the majority voting runs the risks of the well-known 
voting paradox, whereby, depending on the decision makers' individual rankings of the 
profiles, the order the ambiguities are voted on can determine the resolutions that are 
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derived. To avoid this possibility, were it likely, the decision makers should instead be 
required to reach a consensus on their pairwise rankings. 

Logically, three mutually exclusive and exhaustive answers to the above question are 
5 possible: (1) __21 is strictly preferred to _12 or (2) _12 is strictly preferred to 21 or (3) 
they are equally preferred (indifference between _2 1 and _12). 

Notationally, these three preferences can be represented as (1) _21 > _12 or (2) _12 > _21 
or (3) _21 = _12, corresponding to (1) bl + cl > bl + c2 or (2) bl + c2 > b2 + cl or (3) 
10 bl + cl = bl + c2 (where, as usual, ">" is "strictly greater than" and is "equal to"). 

Weak preferences, for example, _21 is at least as preferred as _12 (notationally, _21 £ 
_12), are also a logical possibility. However strict preferences are more useful from a 
practical perspective. 

The decision makers might, quite naturally, protest that their answer to the above 
question ("Given two alternatives that are identical with respect to criterion a, which has 
the greater priority, _21 or _12?") depends on whether criterion a is rated *V or '2' (for 
both alternatives). Nonetheless, such distinctions are precluded by the internal logic of 
APSs. If the decision makers will not answer the question as it is posed then, in effect, the 
APS itself will answer it by default, as the ambiguity will eventually be implicitly 
resolved by the other explicitly resolved ambiguities chosen by the decision makers. 

More specifically, for example, if the decision makers were to decide that 121 > 1 12 — 
25 corresponding to bl + cl > bl + c2 — then this implies 221 > 212, and vice versa. And 
analogously for bl +c2>b2 + cl andM + c2 = 62 + cL 

The key word above is "(/", as clearly neither inequality holds intrinsically. Therefore a 
value judgement is required to resolve ambiguity (1) bl + cl w bl + c2: either 121 > 1 12 
30 andlll > 212 or 112 > 121 andlll > 111 or 121 - 112 and 111 = 212. By virtue of the 
laws of arithmetic, if one inequality or equality holds then the other must too; if one does 
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not then neither does the other. It is impossible to have one half of either proposition 
without the other. 

Accordingly the question above could be rephrased, in essence, as: "Which one of the 
5 following three possible rankings of two alternatives do you prefer, (1) 121 > 112 and 
221 > 212 or (2) 112 > 121 and 212 > 221 or (3) 121 = 1 12 and 221 = 212?" 

Continuing with the example, suppose that in fact the decision makers resolve ambiguity 
(1) by choosing _21 > _12 (in other words 121 > 112 and 221 > 212), corresponding to 
10 b2 + cl > bl + c2. Two alternative, but equivalent, approaches are available for 
identifying the implicitly resolved ambiguities. 

Although these approaches — hereinafter referred to as 'Approach V (of which there are 
two variants) and 'Approach 2' — differ in the means and the sequence in which the 
15 implicitly resolved ambiguities are identified, both generate the same list of explicitly 
resolved ambiguities, from which the point values are derived at Step 3 described below. 

However because Approach 1 becomes relatively unwieldy, and therefore more resource 
intensive to implement, for more criteria and categories than the exemplar APS with x = 3 
20 and y = 2, the particularly preferred embodiment of the computer program of the 
invention is preferably based on Approach 2. 

Approach 1 may be summarised as follows. After a given ambiguity is explicitly resolved 
by the decision makers, all other ambiguities that are implicitly resolved as corollaries are 
25 immediately identified by adding appropriate inherent inequalities and/or other explicitly 
resolved ambiguities (inequalities or equalities). Then another (unresolved) ambiguity is 
explicitly resolved by the decision makers and all its corollaries are identified. The 
process is repeated until all ambiguities have been resolved, either explicitly or implicitly. 

30 Thus a corollary of the explicit resolution of ambiguity (1) as b2 + cl > bl + c2 (decided 
by the decision makers, as explained earlier) is the implicit resolution of ambiguity (4) a2 
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+ b2 + cl vs al + M + c2. This is revealed by adding the inherent inequality a2 > al and 
b2+c\>b\ + c2: (a2>al) + (£2 + cl > M + c2) = (a2 + Z>2 + cl > al + M + c2). 

Although this addition is mathematically legitimate, its theoretical validity in the context 
5 of APSs rests on the assumption that the decision makers are logically consistent in the 
sense that their pairwise profile rankings are transitive. Transitivity means in general that 
if alternatives^ >B andB > C, then ,4 > C. 

Thus, in the present example, the explicit resolution of ambiguity (1) as b2 + cl > bl + c2 
10 corresponds to 121 > 112 and 221 > 212, as discussed earlier. Moreover 212 > 112 
because a2> a\. Therefore, assuming profile rankings are transitive, 221 > 212 and 212 
> 112 implies 221 > 112 — corresponding to a2 + b2 + cl > al + bl + c2, as was 
revealed above by adding a2 > al and b2 + cl > bl + c2. 

15 As there are no other corollaries at this point, the next ambiguity on the list, in this case 
ambiguity (2) a2 + cl vs al + c2, can be presented to the decision makers to resolve via 
an analogous question to the first one: "Given two alternatives that are identical with 
respect to criterion b 9 which has the greater priority, 1_2 or 2_1?" 

20 Suppose the decision makers answer 1_2 > 2 1, corresponding toal +c2>a2 + cl.A 
corollary is the implicit resolution of ambiguity (6) al + b2 + c2 vs a2 + bl + cl, as 
revealed by adding inherent inequality b2 > bl toal + c2> a2 + cl: (b2 > bl) + (al + c2 
>a2 + cl) = (al+Z>2 + c2>a2 + M +cl), 

25 In addition, both (1) b2 + cl > bl + c2 and (2) al + c2 > al + cl implicitly resolve 
ambiguity (3) a2 + M vs al + £>2, as revealed by their addition: (b2 + cl > bl + c2) + (al 
+ c2 > a2 + cl) = (al + b2>a2 + bl) (after cancelling the c terms). 

As for the addition of inherent inequalities and explicitly resolved ambiguities (as above), 
30 this addition is also justified by the assumption that the pairwise profile rankings of the 
decision makers are transitive. 
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Thus, in the present example, the explicit resolution of ambiguity (1) as 62 + cl > 61 + c2 
corresponds to 121 > 1 12 and 221 > 212, and (2) al + c2 > a2 + cl corresponds to 1 12 > 
211 and 122 > 221. Given 121 > 112 and 112 > 211, then, by transitivity, 121 > 211 is 
5 implied. Likewise, given 122 > 221 and 221 > 212, then 122 > 212 is implied. Both 121 

> 21 1 and 122 > 212 correspond to al + b2 > a2 + 61, as was revealed above by adding 
(1) 62 + cl > 61 + c2 and (2) al + c2 > a2 + cl. 

Thus from just two explicit decisions to resolve ambiguities (1) and (2), another three 
10 ambiguities (3, 4 and 5) are implicitly resolved, so that five of the six ambiguities are 
resolved. 

The remaining ambiguity, ambiguity (5) a2 + 61 + c2 vs al + b2 + cl, must be explicitly 
resolved by the decision makers, via a question that is conceptually simpler than the two 
15 earlier ones: "Which alternative has the greater priority, 212 or 121?" Suppose the 
decision makers answer 212 > 121, corresponding to a2 + bl + c2 > al + b2 + cl . 

The system is now fully specified as: (1) 62 + cl > 61 + c2, (2) al + c2 > a2 + cl, (3) al 
+ 62 > a2 + 61, (4) a2 + 62 + cl > al + 61+ c2, (5) a2 + 61 + c2 > al +62 + cl and (6) 
20 al + 62 + c2 > a2 + 61 + cl — as well as the inherent inequalities al > al, 62 > 61 and 
c2 > cl. Of inequalities (1) to (6), only three (1, 2 and 5) were explicitly resolved by the 
decision makers, with the other three (3, 4 and 6) implicitly resolved as corollaries. 

With respect to the restricted group of Alternatives 1 to 4 in the example mentioned 
25 earlier, corresponding to three 2 nd -degree ambiguities and one S^-degree ambiguity, the 
fully specified system is: (1) 62 + cl > 61 + c2, (2) al + c2 > a2 + cl, (3) al + 62 > a2 + 
61 and (6) al + 62 + c2 > a2 + 61 + cl — as well as the inherent inequalities a2 > al, 62 

> 61 and c2 > cl. These inequalities are arrived at (decided by the decision makers) via 
the same process as explained above for fully calibrating the APS, except that only 

30 inequalities (1) and (2) must be explicitly resolved by the decision makers, with the other 
two (3 and 6) implicitly resolved as corollaries. 
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Finally, in general but not in the present example, any explicitly resolved ambiguities that 
are themselves corollaries of other explicitly resolved ambiguities can be removed from 
the list from which the point values are derived at Step 3 (below). This is because only 
5 independent inequalities/equalities are required for deriving point values. 

A variant of the approach outlined above — in effect, its converse — is the identification 
of the sufficient (but not necessary) conditions for implicitly resolving ambiguities, in 
terms of (other) resolved ambiguities. Ambiguities are either explicitly resolved by the 
10 decision makers or implicitly resolved when their sufficient conditions are met. Any 
ambiguities whose sufficient conditions are not met must therefore be resolved explicitly 
by the decision makers, until all ambiguities have been resolved, either explicitly or 
implicitly. 

15 Thus in the exemplar APS with x = 3 and y = 2, ambiguities (4), (5) and (6) have four 
sufficient conditions each in terms of resolved 2 nd -degree ambiguities. Specifically, 
ambiguity (4) is (implicitly) resolved as al + bl + cl > al + bl + c2 if at least one of the 
following inequalities holds: al + cl > al + c2 or a2 + cl = al + c2 (in both cases 
because bl > bl) or b2 + cl > bl + cl or bl + cl = bl + cl (in both cases because al > 

20 al). 

These and analogous sufficient (but not necessary) conditions for ambiguities (5) and (6) 
to be implicitly resolved as al + bl + cl > al + bl + cl and al + £2 + c2 > al + M + cl 
respectively are listed in Figure 10. No sufficient conditions exist for the opposite 
25 resolutions of the three ambiguities nor for equalities (that is, not for RHS > LHS nor 
LHS = RHS) in terms of resolved 2 nd -degree ambiguities. 

It is then simply a matter of comparing these conditions against resolved ambiguities (1), 
(2) and (3) (arrived at via Approach 1 explained above). Accordingly, as identified via 
30 shading in Figure 10, (1) bl + cl > bl + c2 implicitly resolves ambiguity (4) as al + bl + 
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cl > al + bl + c2, and both (2) al + c2 > a2 + cl and (3) al + bl > a2 + 61 implicitly 
resolve ambiguity (6) as al + b2 + c2 > a2 + 61 + cl. 

These are the same 3 rd -degree resolutions as were revealed earlier via the explicit 
5 resolutions of ambiguities (1) and (2) and their additions to inherent inequalities and each 
other respectively. As such they reflect, as before, pairwise profile rankings that are 
transitive. 

Finally, as before, ambiguity (5) remains to be explicitly resolved, and then the system is 
10 fully specified. 

This variant of Approach 1 generalises for APSs with higher-degree ambiguities. Their 
sufficient conditions are in terms of both individual lower-degree resolved ambiguities 
and combinations of them, of which there may be many. For example, the sufficient 
15 conditions for the resolution of a S^-degree ambiguity are in terms of combinations of 
2 nd -degree resolved ambiguities, S^-degree resolved ambiguities and 4 th -degree resolved 
ambiguities. 

However, sufficient conditions for an ambiguity of a given degree can also be identified 
20 in terms of resolved ambiguities of the same, or even higher, degrees. In the present 
example, sufficient conditions can also be identified for resolving 2 nd -degree ambiguities 
in terms of other resolved 2 nd -degree ambiguities. For example, a sufficient condition for 
resolving ambiguity (3) a2 + bl vs al + b2 as al + b2 > a2 + bl is (1) b2 + cl > bl + c2 
arid (2) al + c2 > a2 + cl, corresponding to (b2 + cl > bl + c2) + (al + c2 > al + cl), as 
25 explained earlier. 

This means that in general it is difficult to enumerate and check all possible sufficient 
conditions (to ensure that none are missed). Therefore, this variant may be supplemented 
by other methods, such as the first variant of Approach 1 explained above. 

30 
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Finally, with respect to both variants of Approach 1, because ambiguities (1) to (3) are 
not independent, both the order and the manner in which they are resolved affects the 
number of explicit value judgements that are required. The maximum number of explicit 
rankings required is four and the minimum is two. 

5 

For example, if inequality (3) had been decided before inequalities (1) and (2) (instead of 
after, as above), then all three ambiguities (as well as ambiguity 5) would have had to 
have been resolved explicitly. This is because inequality (3) al + 62 > a2 + 61 cannot be 
added to (1) b2 + cl > bl + c2 or (2) al + c2 > a2 + cl to obtain the other inequality, and 
10 yet inequalities (1) and (2) imply (3). For ambiguities (4) to (6), on the other hand, one 
and only one must be resolved explicitly. 

Alternatively, for example, had ambiguity (1) been resolved as b2 + cl = bl + c2 instead 
of b2 + cl > bl + c2 (indifference rather than strict preference), and ambiguity (2) 
15 resolved as al + c2 > a2 + cl (as above), then no other explicit resolutions would have 
been necessary. 

This can be confirmed by noting that (1) b2 + cl = bl + c2 implies both (4) a2 + b2 + cl 
> al + bl + c2 and (5) a2 + bl + c2 > al + b2 + cl. In other words (62 + cl = bl + c2) + 
20 (62 > 61) for both of them and (2) al + c2 > a2 + cl implies (6) al + 62 + c2 > a2 + 61 + 
cl (as above) and (62 + cl = 61 + c2) + (al +c2>a2 + cl) implies (3) al + 62 > a2 + 
61. Clearly, the point values derived from this system of equations and inequalities would 
be different to the point values derived earlier. 

25 Similarly the system would be completely specified by the explicit resolution of 
ambiguities (1) and (6) as 61 + c2 > 62 + cl and a2 + 61 + cl > al + 62 + c2 only. Of 
particular interest is: (a2 + 61 + cl > al + 62 + c2) + (62 > 61) = (2) (a2 + cl > al + c2); 
and (a2 + 61 + cl > al + 62 + c2) + (c2 > cl) = (3) (a2 + 61 > al + 62). This illustrates 
the fact that it is not necessary to resolve lower-degree (here 2 nd -degree) ambiguities 

30 before implicitly resolving higher- degree (here S^-degree) ambiguities. The process can 
be reversed, as illustrated here. 

175363-5 



WO 2004/111906 



PCT/NZ2004/000121 



35 



The 'additions' variant of Approach 1 involves maintaining a list of all possible 
5 inequalities/equalities that result from the addition of each explicitly resolved ambiguity 
with each other explicitly resolved ambiguity as well as with the inherent inequalities, 
and with every sum generated, and each of these sums with each other recursively until 
all possible additive combinations are exhausted. Each newly generated ambiguity is 
'checked off against the list of implicitly resolved ambiguities: if it is not on the list then 
10 it is yet to be resolved 

Similarly, the 'sufficient conditions' variant of Approach 1 involves managing a list of all 
implicitly resolved ambiguities, identifying all possible sufficient conditions of lower 
degrees, of which there may be a great many combinations, and identifying ambiguities 
15 implied by the same or higher degree explicitly resolved ambiguities (by some other 
means). 

Approach 2, on the other hand, which is explained below, is more efficient and is 
therefore the preferred method for fully calibrating APSs, and also for ranking a restricted 
20 group of alternatives or choosing the 'best' alternative from the group, used in the 
computer program of the invention. 

In summary, Approach 2 involves testing ambiguities individually for whether or not they 
are implicitly resolved as corollaries of the explicitly resolved ambiguities (that were 
25 resolved earlier). If a given ambiguity is identified as having been implicitly resolved 
then it is deleted. If instead it is not implicitly resolved then it must be explicitly resolved 
by the decision makers. The process is repeated until all ambiguities have been identified 
as having been implicitly resolved or they are explicitly resolved. 

30 Thus, with reference to the exemplar APS with jc= 3 and y =2, after the decision makers 
resolve ambiguity (1) as 62 + cl > 61 + c2 as described earlier, the next ambiguity on the 
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list ((2) al + c2 vs a2 + cl) is tested for whether or not it is implicitly resolved as a 
corollary of 62 + cl > bl 4- c2, as well as the inherent inequalities al > al, b2 > bl and 
c2>cl. 

5 Here, as in the computer program of the invention, this and subsequent tests can be 
performed via linear programming. In effect, this test is performed by asking the 
following two hypothetical questions (of the method, not the decision makers). 

Hypothetical Question 1: If it were the case that the ambiguity in question [here (2) al + 
10 c2 vs a2 + cl] had been implicitly resolved as LHS > RHS (i.e., al + c2 > a2 + cl), then 
does a solution exist to the system comprising this hypothetical inequality and the (actual) 
explicitly resolved inequalities/equalities [here (1) b2 + cl > bl + c2] — as well as the 
inherent inequalities [here a2 > al, b2 > bl and c2 > cl]? (Yes or No?) 

1 5 if the answer is No — and therefore it would not be theoretically possible for the decision 
makers, if they wanted to, to explicitly resolve the ambiguity in question as LHS > RHS 
(here al + c2 > a2 + cl) — then it must be true that either RHS > LHS or LHS = RHS 
(i.e., either a2 + cl > al + c2 or al + c2 = a2 + cl). This implies that the ambiguity in 
question (here ambiguity 2) has been implicitly resolved (i.e., as either a2 + cl > al + c2 

20 or al + c2 = a2 + cl). Hence it is of no further use and can be deleted. 

If instead the answer to Question 1 is Yes — and therefore it would be theoretically 
possible for the decision makers, if they wanted to, to explicitly decide LHS > RHS (here 
al+c2>a2 + cl) — then the following second hypothetical question is asked. 

25 

Hypothetical Question 2: If it were instead the case that the ambiguity in question (here 
ambiguity 2) had been implicitly resolved as RHS > LHS (that is, a2 + cl > al + c2), 
then does a solution exist to the system comprising this hypothetical inequality and the 
(actual) explicitly resolved inequalities/equalities [here (1) b2 + cl > bl + c2, as before] 
30 — as well as the inherent inequalities [here a2 > al, b2 > bl and c2 > cl]? (Yes or No?) 
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If tilt answer is No — and therefore it would not be theoretically possible for the decision 
makers, if they wanted to, to explicitly resolve the ambiguity in question as RHS > LHS 
(here al + cl > al + c2) — then it must be true that either LHS > RHS or LHS = RHS 
(either al + c2 > al + cl or al + c2 = a2 + cl). This implies that the ambiguity in 
5 question (here ambiguity 2) has been implicitly resolved, in this case asal+c2><z2 + cl 
or al + c2 = a2 + cl . Hence it is of no further use and can be discarded. 

if instead the answer to Question 2 is Yes then it must be inferred that as well as it being 
theoretically possible for the decision makers, if they wanted to, to explicitly resolve the 
10 ambiguity in question as LHS > RHS (here al + c2 > al + cl) from Question 1 that it is 
also theoretically possible for them to resolve it as RHS > LHS (al + cl > al + c2). This 
implies that the ambiguity in question (here ambiguity 2) is not implicitly resolved as a 
corollary of the explicitly resolved ambiguities, and therefore it must be explicitly 
resolved by the decision makers. 

15 

In the case of ambiguity (2) al + c2 vs al + cl the answers to Questions 1 and 2 are Yes 
and Yes, and so the ambiguity should be presented to the decision makers for them to 
explicitly resolve. As for the earlier demonstration of Approach 1, suppose it is decided 
al+c2>a2 + cl. 

20 

The next ambiguity on the list — (3) al + bl vs al + bl — is then tested via the same 
process as outlined above. This time, though, the list of (actual) explicitly resolved 
inequalities/equalities against which (3) al + bl > al + bl (i.e., LHS > RHS, as for 
Question 1) and then, if necessary, (3) al + bl > al + bl (RHS > LHS, as for Question 2) 
25 are tested comprises (2) al + c2 > al + cl as well as (1) bl + cl > bl 4- c2 (as before). 

t 

Thus the list of explicitly resolved inequalities/equalities is continually updated — 
including in general but not in the present example, as for Approach 1, the identification 
of any explicitly resolved inequalities/equalities that are themselves corollaries of others 
30 on the list. 
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The process is repeated for all ambiguities until all of them have been identified as having 
been implicitly resolved or they are explicitly resolved by the decision makers. This can 
be summarised for the present example as follows. 

5 For ambiguity (3) the answer to Question 1 is No, and therefore this ambiguity is 
identified as having been implicitly resolved as a corollary of the explicitly resolved 
ambiguities. For ambiguity (4), the answer to Question 1 is Yes but the answer to 
Question 2 is No, and therefore this ambiguity is identified as having been implicitly 
resolved. 

10 

For ambiguity (5) the answers to both questions are Yes, implying that this ambiguity has 
not been implicitly resolved, and so it must be explicitly resolved by the decision makers: 
as a2 + bl + c2 > al + b2 + cl, as for the demonstration of Approach 1. Finally, for 
ambiguity (6) the answers are Yes and No, and therefore this ambiguity is also identified 
15 as having been implicitly resolved. 

Thus, of the six ambiguities, three had to be explicitly resolved by the decision makers — 
(1) bl + cl > bl + c2, (2) al + c2 > a2 + cl and (5) a2 + bl + c2 > al + b2 + cl (the 
same three as for Approach 1) — with the other three (3, 4 and 6) were identified as 
20 having been implicitly resolved as corollaries. 

The final step of the method (Step 3) of the invention involves simultaneously solving the 
system of (independent) explicitly resolved ambiguities (inequalities and equalities) and 
inherent inequalities to obtain the point values and/or a ranking of the restricted group of 
25 alternatives or the identification of the 'best* one. This step of the method is represented 
in Figure 5 at 530. 

As described above, any explicitly resolved ambiguities that are themselves corollaries of 
other explicitly decided inequalities are removed, because only independent 
30 inequalities/equalities are required for deriving the point values. 
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For the example of fully calibrating an APS with x = 3 mdy = 2, one solution for (1) 62 
+ cl >61 + c2, (2)al + c2>a2 + cl and (5)a2 + 61 + c2 > + 62 + cl anda2>al, 62 
> 61 and c2 > cl is: al = 0, a2 = 2, 61 =0, 62 = 4, cl =0 and c2 = 3. These point values 
produce ranking #9 in Figure 4. 

5 

Similarly, with respect to the restricted group of Alternatives 1 to 4 in the example 
mentioned earlier, these point values produce the ranking: 122 (first), 221, 212 and 211 
(last). 

10 The process for ranking a restricted group of alternatives or choosing the 'best' 
alternative from the group is now discussed in more detail. It is preferable to integrate it 
with the process for fully calibrating an APS, as described above, as the ranking of the 
restricted group or identification of the 'best' alternative may occur before the APS is 
fully calibrated. 

15 

After the restricted group of alternatives is specified by the decision makers, the 
ambiguities for these alternatives are generated using the 'simple method for generating 
ambiguities' described above. In essence, this involves pairwise comparisons of 
alternatives and the cancellation of variables common to both alternatives. 

20 

When these ambiguities are resolved by the decision makers, the restricted group of 
alternatives will have been ranked such that any further decisions made for the purpose of 
fully calibrating an APS will not affect the ranking of the restricted group of alternatives. 
These ambiguities for the restricted group of alternatives are tested for resolution 
25 whenever the decision makers explicitly resolves an ambiguity in the same manner as for 
the other ambiguities, as described above. 

However, the 'best' or top-ranked alternative may be able to be determined before the 
restricted group of alternatives is ranked. There are two possible approaches to this, 
30 which are referred to below as Approach 3 and Approach 4. They are now explained in 
turn. 
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For Approach 3, as the decision makers explicitly resolve ambiguities, each alternative is 
compared with every other to determine whether it would be infeasible for it to be ranked 
below the other alternatives, given the explicitly resolved ambiguities and the inherent 
5 inequalities. This test is analogous to that described earlier in terms of Hypothetical 
Questions 1 and 2 of Approach 2 above, as is now illustrated in the present context 

Hypothetical Question 1: Does a solution exist to the system comprising the (actual) 
explicitly resolved inequalities/equalities as well as the inherent inequalities and the 
10 proposition that 'Alternative A* > 'Alternative B\ If the answer is No, and therefore it 
would not be theoretically possible for 'Alternative A* to be ranked above or equal to 
'Alternative B\ then it must be true that 'Alternative A' is ranked below 'Alternative B\ 
Hence 'Alternative B' is not a 'best' alternative. 

15 If instead the answer to Question 1 is Yes, and therefore it would be theoretically possible 
for 'Alternative A' to be ranked above or equal to 'Alternative B\ then the following 
second hypothetical question is asked. 

Hypothetical Question 2\ Does a solution exist to the system comprising the (actual) 
20 explicitly resolved inequalities/equalities as well as the inherent inequalities and the 
proposition that 'Alternative B' > 'Alternative A'. If the answer is No, and therefore it 
would not be theoretically possible for 'Alternative B' to be ranked above or equal to 
'Alternative A\ then it must be true that 'Alternative B' is ranked below 'Alternative A*. 
Hence 'Alternative B' is not a 'best 5 alternative. 

25 

If instead the answer to Question 2 is Yes, and therefore it would be theoretically possible 
for 'Alternative A' to be ranked above or equal to 'Alternative B\ then the relationship 
between these two alternatives is either not yet known or they are equal. 

30 After all the alternatives have been compared with each other, a final test is performed 
among those alternatives that have not been identified as not being a 'best' alternative. 
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These potentially 'best' alternatives are pairwise compared with every other to determine 
if it is possible that they are not equal, in a fashion analogous to the above (if it is 
possible that 'Alternative A' < 'Alternative B', or that 'Alternative B' < 'Alternative A'). 
If it is possible that these potentially 'best 5 alternatives are not equal then there is no 
5 clearly 'best' alternative. 

If instead all the potentially 'best' alternatives are found to be equally ranked, or if there 
is only one, the highest ranking alternative or alternatives will not change with further 
resolution of ambiguities, and if finding the 'best 5 alternative is the objective, the process 
1 0 can stop before the APS is fully calibrated. 

An alternative approach, known as Approach 4, to finding the 'best' alternative that 
extends more easily to finding the best t alternatives among the restricted group of 
alternatives is as follows. 

15 

First, a solution to the system of inherent inequalities and resolved ambiguities is found, 
resulting in a set of points values for each category. Second, these points values are used 
to score each alternative in the restricted group of alternatives. Third, the group is sorted 
by each alternative's score. Fourth, the unresolved ambiguities representing an ambiguity 
20 involving any of the top t alternatives (the t alternatives with the highest scores) and the 
other alternatives are sought. 

If no such ambiguities are found, the top t alternatives are ranked ahead of the other 
alternatives. This relationship between the top / alternatives and the other alternatives will 
25 not change with further resolution of ambiguities and so the 'best 5 t alternatives have 
been found. If finding the 'best 5 1 alternatives is the objective, the process can stop before 
the APS is fully calibrated. 

Fifth, if the 'best 5 t alternatives were found, the unresolved ambiguities rqpresenting 
30 ambiguities involving alternatives in the top t alternatives are identified. If no such 
ambiguities are found, the best t alternatives are also ranked among each other, and this 
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relationship will not change with further resolution of ambiguities. If finding the 'best' t 
alternatives and ranking these / alternatives is the objective, the process can stop before 
the APS is fully calibrated. 

5 The overall method explained above may be broadly summarised as follows. First, 
generate the ambiguities of the APS that is to be calibrated and/or the ambiguities of the 
restricted group of alternatives that are to be ranked or the 'best' chosen as shown at 510 
in Figure 5. 

10 Then explicitly resolve the ambiguities via the value judgements of the consulted decision 
makers, while identifying all other ambiguities that are implicitly resolved as corollaries, 
until all ambiguities are resolved. This step is illustrated at 520 in Figure 5. Although, in 
theory, the implicitly resolved ambiguities can be identified via. Approach 1 ox Approach 
2 (both explained above), the latter is more efficient and therefore it is the particularly 

1 5 preferred method for the computer program of the invention, as explained below. 

Finally, simultaneously solve the system of explicitly resolved ambiguities (that is, 
inequalities and equalities) and inherent inequalities to obtain the point values for the 
APS and/or a ranking of the restricted group of alternatives or the identification of the 
20 'best' one. This step is illustrated at 530 in Figure 5. Although, in theory, the 'best' or 
top-ranked alternative can be identified via Approach 3 or Approach 4 (both explained 
above), the latter is more efficient and therefore it is the particularly preferred method for 
the computer program of the invention, as explained below. 

25 The property (assumption) that the profile rankings of the decision makers are transitive 
(or logically consistent), enables the number of ambiguities that must be explicitly 
resolved at 520 to be minimised, with the remainder emerging implicitly as corollaries. 

For APSs with more criteria and categories than the exemplar with x = 3 and y = 2, 
30 minimising the number of explicitly resolved ambiguities, and accordingly the number of 
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value judgements required from the decision makers, is a significant practical advantage 
of our method. 

A prefeired computer program for implementing the method described above will now be 
5 described more particularly. The computer program of the invention comprises, in broad 
terms, five steps, as explained in turn below. An overview of the five steps is illustrated 
in Figure 1 1 . The computer program also includes several features that arise from, but are 
supplementary to, the method described above, 

10 Before starting the program, the user — who may be one of the decision makers 
consulted to resolve the ambiguities, or, alternatively, he or she may be a facilitator of the 
calibration process — must have chosen the criteria and categories of the APS that is to 
be calibrated and must have ranked each criterion's categories. 

15 At Step 1, when the program begins, the user may be asked to enter a title for the APS 
and the criteria and their categories, and to rank the categories for each criterion. 
Typically this would be achieved via a user interface and user input/output devices and 
the data received stored to data memory for later processing- The criteria and categories 
must be labelled in terms of the variables (e.g. 'al\ *a2 9 *a3\ etc.) and verbally 

20 described in preparation for their later presentation to the decision makers. 

The user is also given the opportunity of listing theoretically impossible combinations of 
criteria and categories that partially or fully specify profiles — known as the To Be 
Excluded list — and that are therefore to be used to cull ambiguities that are generated by 
25 the program. This list may be left empty if the user wishes. 

The To Be Excluded list may also include alternatives that are regarded by the user as 
effectively irrelevant to the decision situation. Such 'irrelevant alternatives' could include 
ones that, regardless of the outcome of the APS calibration exercise, will always be 
30 dominated by the 'relevant alternatives' that, because only they (the relevant alternatives) 
will be practically affected by the overall ranking produced, are the focus of the user. For 
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example, when patients are being ranked for treatment, if only those in the top portion of 
the ranking will receive the treatment, then those in the bottom portion are effectively 
irrelevant. 

5 Irrelevant alternatives are easily identified by the computer program calculating for each 
theoretically possible alternative the number of alternatives that are unambiguously 
ranked above it (in other words, that dominate the alternative). This statistic (denoted 
below by d) for each alternative is the product across all criteria of the difference for each 
criterion between the lowest-ranked possible category and the actual category for the 
10 alternative plus one, minus one (from the product). 

Thus the highest possible rank for a given alternative is d + 1. If the user's focus is the 
top c ranked alternatives, then all alternatives for which d > c can be excluded and the 
remaining alternatives scored. The user is given the opportunity of specifying c, thereby 
1 5 identifying irrelevant alternatives to be on the To Be Excluded list 

The user is also given the opportunity of specifying a restricted group of alternatives in 
terms of combinations of criteria and categories that are to be ranked and/or the 'best' 
chosen — known as the Loaded Alternatives list. This list may be left empty if the user 
20 wishes. 

After the program is initialised, using equation (3) above it calculates the number of 
unique ambiguities to be resolved to fully calibrate the APS. This can be reported to the 
user and used to estimate whether the system can be solved in an acceptable amount of 
25 time given the computing resources that are available. 

The user is also given the opportunity of estimating the number of decisions that will be 
required to fully calibrate the APS by instructing the program to simulate the calibration 
process that is described in detail below. This number of decisions depends on the 
30 numbers of criteria and categories specified by the user. It also depends on the 
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ambiguities explicitly resolved by the user and whether the user wishes to fully or 
partially calibrate the APS. 

The program simulates the user's overall rankings of all possible alternatives that are 
represented by the criteria and categories by (repeatedly) randomly drawing the point 
values and ranking the alternatives by their overall value scores. Equipped thus with 
simulated overall rankings of alternatives and hence pairwise rankings, the computer 
program explained below is simulated and the number of decisions required to fully 
calibrate the APS, or to resolve all ambiguities up to a given degree, is counted. These 
statistics are available to the user and can be used to estimate the amount of effort 
required of him or her to calibrate the APS. 

The ambiguities for the restricted group of alternatives on the Loaded Alternatives list are 
generated using the 4 simple method for generating ambiguities' described above. In 
essence, this involves pairwise comparisons of alternatives and the cancellation of 
variables common to both alternatives. These ambiguities are placed on the Alternatives' 
Ambiguities list, which is used to determine if the restricted group of alternatives has been 
ranked such that further decisions will not affect the ranking. Furthermore, all alternatives 
on the Loaded Alternatives list are placed on the Best Alternatives list, which is used to 
determine whether the best alternative (or alternatives) have been determined such that 
further decisions will not affect their top (or top equal) ranking. These steps are illustrated 
in Figure 16. The process for managing these lists will be discussed later below. 

Step 2 of the program involves generating the ambiguities for the APS using the 'efficient 
ambiguities generator' described earlier. An overview of this step is illustrated in Figure 
12. 

For simplicity and efficiency, ambiguities axe generated for the APS by the efficient 
ambiguities generator one degree at a time, beginning with the 2 nd -degree, as determined 
by the value of a control variable — known as the Current Degree variable (see Figure 
11) — which is initially set to 2. 
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As each ambiguity is generated it is checked for whether or not it is theoretically 
possible, given the To Be Excluded list of partially or fully specified profiles. If the 
ambiguity is theoretically possible it is then tested via Approach 2 of Step 2 of the 
5 method of the invention explained earlier for whether or not it is implicitly resolved by 
the explicitly resolved ambiguities. An overview of the procedure is illustrated in Figure 
13. Note though that the Test list generated in Figure 13 need only be generated once for 
each batch of ambiguities being tested. 

10 This and similar tests may be performed via linear programming, which is described 
below. It will, however, be appreciated by those skilled in the art that the methodology of 
the invention may be implemented using any appropriate programming method 

If an ambiguity is found to be implicitly resolved then it is discarded; otherwise it is 
15 added to a list known as the To Be Resolved list. (Note that when the 2 nd -degree 
ambiguities are generated, none will be implicitly resolved because none have yet been 
explicitly resolved.) 

Step 3 of the program is to present the To Be Resolved list of unresolved ambiguities for 
20 the current degree to the user. An overview of this step and Step 4 (explained below) is 
illustrated in Figure 14. 

The user can choose to view the ambiguities either in equation form (for example, al + 

b2 vs a4 + M) or symbolically (for example, 21 or 41 ), which are ordered 

25 either randomly or by their criteria and categories. The user is invited to select an 
ambiguity for the purpose of explicitly resolving it The selected ambiguity is 'translated' 
verbally in terms of the criteria and category descriptions. 

If the user desires, he or she may skip this particular ambiguity and select another one, or 
30 she may resolve it by clicking one of three buttons labelled (in essence): "LHS greater" or 
"RHS greater 55 (i.e., < or RHS preferred) or "LHS and RHS equal". If "RHS greater 
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(preferred)" is chosen, for consistency, the LHS and RHS of the resolved ambiguity are 
switched and stored as RHS > LHS. 

Alternatively, the user may indicate that one or both of the alternatives is theoretically 
5 impossible and this is then added to the To Be Excluded list. Any ambiguities on the To 
Be Resolved list matching a new impossible alternative may be deleted. 

The system can also permit weak inequalities (such as "LHS greater than or equal to 55 and 
"RHS greater than or equal to"), however this means that some ambiguities will later be 
10 partially (weakly) solved and so some buttons in the user interface must be disabled when 
the ambiguity is selected. Note that in any resulting APS the result will be either "greater 
than" or it will be "equal to" but it will not be both. The system can also be designed so 
that only strong inequalities and no equalities are permitted, thereby producing strict 
profile rankings only. 

15 

Step 4 is for the program to remove the explicitly resolved ambiguity (as above) from the 
To Be Resolved list and add it — as either an inequality or equality (depending on how 
the ambiguity was resolved) — to the list of explicitly resolved ambiguities, known as the 
Explicitly Resolved list. An overview of this step and Step 3 is illustrated as Figure 14. 

20 

Each inequality/equality on the Explicitly Resolved list is then tested as to whether or not 
it is implied by the others on the list. Any found to be implied may be marked as being 
'redundant' and hereinafter ignored, but they are not deleted because they may be re-used 
later if any explicitly resolved ambiguities are later revised by the decision makers 
25 (explained below). 

All ambiguities on the To Be Resolved list are then tested for whether or not their 
resolution is implied by the (non-redundant) inequalities/equalities on the Explicitly 
Resolved list, and if so they are deleted from the To Be Resolved list This test was 
30 explained earlier in terms of Hypothetical Questions 1 and 2 of Approach 2 of the 
method's Step 2 and is illustrated in Figure 13. 
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In essence, the test involves finding whether or not a solution (in terms of feasible point 
values) exists to a system comprising the explicitly resolved inequalities/equalities and 
inherent inequalities, and each of the possible hypothetical inequalities in turn (that is 
5 LHS > RHS and RHS > LHS) corresponding to the ambiguity in question. This and the 
earlier tests based on determining the existence of solutions are performed via linear 
programming (LP) with inequality and equality constraints. LP is discussed in more detail 
in the final section below. 

10 Next, the ambiguities on the Alternatives ' Ambiguities list are tested and removed in the 
same way as the To Be Resolved list, as referred to in Figure 14. If the Alternatives' 
Ambiguities list is not then empty, the process continues; however, once the Alternatives' 
Ambiguities list is empty, all alternatives have been ranked. The system then solves the 
point values, as referred to as Step 5 in Figure 1 1, to obtain the scores for the alternatives, 

15 and then presents the ranked alternatives to the user. An overview of this process is 
illustrated in Figure 17. The user can continue to calibrate the APS if he or she wishes, 
but the ranking of the alternatives will not change unless the user undoes some prior 
decisions or introduces new alternatives to the restricted group under consideration. 

20 If the Alternatives' Ambiguities list is not now empty, each alternative on the Best 
Alternatives list is compared with every other, and if it is not possible for an alternative to 
be ranked ahead of or equal to another alternative given the decisions on the Explicitly 
Resolved list and the inherent inequalities, then that alternative is removed from the Best 
Alternatives list. After this processing, the alternatives on the Best Alternatives list are 

25 again compared with each other, and if it is not possible for any of the alternatives to be 
ranked ahead of one of the others, the list is known to contain either one 'best* (i.e., top- 
ranked) alternative, or equally top-ranked alternatives. 

The system then solves the point values, as referred to as Step 5 in Figure 1 1, to obtain 
30 the scores for the alternatives, and thai presents the alternative or alternatives on the Best 
Alternatives list to the user. An overview of this process, which was referred to as 
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Approach 3 in the description of the method of the invention, is illustrated in Figure 18. 
The user can continue to fully calibrate the APS if he or she wishes, but the highest 
ranking alternative or alternatives will not change unless the user undoes some prior 
decisions or introduces new alternatives to the restricted group under consideration. 

5 

If a particular number of 'best' alternatives is required — for example, if the objective is 
to select the top three alternatives — this process can be repeated as many times as 
required after setting aside the top-ranked alternative or alternatives (for late: reporting) 
from the Loaded Alternatives list and regenerating the Best Alternatives list, as illustrated 
10 in Figure 16. The test as to whether it is possible for a given alternative to be ranked 
ahead of or equal to another alternative given the decisions on the Explicitly Resolved list 
and the inherent inequalities is similar to (but different from) that used in determining 
whether an ambiguity is resolved. This test, using linear programming, is discussed 
below. 

15 

However, the particularly preferred way of determining whether a particular number (f) 
of 'best' alternatives has been found is as follows. An overview of this process, which 
was referred to as Approach 4 in the description of the method of the invention, is 
illustrated in Figure 19. 

20 

First, the system is solved as in Step 5 below, establishing a set of points values. Second, 
these points values are used to score each alternative on the Loaded Alternatives list. 
Third, the Loaded Alternatives list is sorted by each alternative's score. Fourth, the 
ambiguities on the Alternatives ' Ambiguities list representing an ambiguity involving any 
25 of the top t alternatives (the t alternatives with the highest scores) on the Loaded 
Alternatives list and the other alternatives on the Loaded Alternatives list are sought. 

If no such ambiguities are found, the top / alternatives are ranked ahead of the other 
alternatives and so the 'best' t alternatives have been found and this result can be reported 
30 to the user. If finding the 'best* t alternatives is the objective, the process can stop before 
the APS is fully calibrated. 
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Fifth, if the 'best' t alternatives were found, the ambiguities on the Alternatives* 
Ambiguities list representing ambiguities involving alternatives in the top t alternatives on 
the Loaded Alternatives list are identified. If no such ambiguities are found, the best t 
5 alternatives are also ranked among each other, and this relationship will not change with 
further resolution of ambiguities, and this result can be reported to the user. If finding the 
'best' / alternatives and ranking these t alternatives is the objective, the process can stop 
before the APS is fully calibrated. 

10 If the To Be Resolved list is not empty, the program returns to Step 3. If instead the To Be 
Resolved list is empty and the Current Degree is equal to the number of criteria (x) in the 
particular APS being calibrated, the program proceeds to Step 5. Alternatively, if the 
Current Degree is less than the APS's number of criteria (and the To Be Resolved list is 
empty), the Current Degree is increased by * V and the program returns to Step 2. 

15 

However, the user is able to stop the process before the APS is fully calibrated as tests 
have shown that the ranking of all hypothetical alternatives after an APS is fully 
calibrated correlates very well with the ranking after only the 2 nd -degree ambiguities have 
been resolved* Such a decision to stop the process would depend on the precision 
20 required of the resulting APS by the user. 

The user may choose at any time to view the explicitly resolved ambiguities for all 
degrees and 'undo' any of them. Explicitly resolved ambiguities that are 'redundant' are 
displayed in a different colour or shade to indicate that undoing them alone will have no 
25 material effect An overview of the 'undo module', as referred to in Figure 11, is 
illustrated in Figure 15. 

When explicitly resolved ambiguities are undone, they are deleted from the Explicitly 
Resolved list. If any non-redundant explicitly resolved ambiguities are undone, the 
30 explicitly resolved ambiguities marked redundant are re-tested for whether or not they are 
still redundant, and marked accordingly. 
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Also, if non-redundant ambiguities are undone, the Current Degree variable is reset to the 
degree of the lowest-degree ambiguity that was undone, the Alternatives' Ambiguities and 
Best Alternatives lists are regenerated as illustrated in Figure 16, and the program restarts 
5 from Step 2 of the computer program as set out above. 

When all the ambiguities for all degrees have been resolved, the list of explicitly resolved 
inequalities/equalities and inherent inequalities is solved via linear programming for the 
point values of the APS. These are presented to users, as well as a range of 'summary 
10 statistics', including the numbers of ambiguities and explicitly resolved ambiguities at 
each degree and, if the user wants them, the inequalities/equalities that were chosen and 
the restricted group of alternatives, if they were specified, and their rankings. 

In the interests of deriving point values that are integers and as low as possible, thereby 
15 maximising their 'user-friendliness', integer programming and an objective function that 
minimises the sum of the variables (for example, al + a2 + a3 4- M + bl + b3 + cl + c2 + 
c3, etc.) may be specified. 

Alternatively, were it desired by users that the point values be normalised to a particular 
20 range for the profile total scores, such as 0 to 100, the derived point values are scaled 
accordingly. However this usually forces some point values to be fractions, which is not 
as 6 user-friendly' as integers. Accordingly both integer and normalised values can be 
calculated and presented. 

25 The user may be given the opportunity of checking the consistency or repeatability of his 
or her decisions with respect to explicitly or implicitly resolved ambiguities. Beginning 
with the 2 nd -degree, ambiguities are generated by the 'efficient ambiguities generator' 
and the above-mentioned derived point values are used to determine their rankings, 
corresponding to the user's explicitly or implicitly resolved ambiguities. The user is then 

30 re-presented with the ambiguities and asked to rank them again. He or she is informed as 
to whether or not the checked decisions agree. If the decisions agree, then they are 
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consistent. If the decisions are contradictory, then they are inconsistent. The numbers of 
agreements and disagreements are counted and reported to the user. 

The user may also be given the opportunity of comparing the current system he or she has 
5 created with another existing and saved system created earlier by the program (provided 
they have the same numbers of criteria and categories). This comparison is in terms of the 
explicitly or implicitly resolved ambiguities, the overall ranking of all possible 
alternatives that are represented by the criteria and categories, and the overall ranking of 
the restricted group of alternatives. The purpose of these comparisons is to gauge the 
10 validity and reliability of the overall user's decisions and the resulting rankings of all 
possible alternatives and the restricted group of alternatives. 

The user may also be given the opportunity of copying the current system and changing 
some explicitly or implicitly resolved ambiguities (while retaining others), or all of them, 
15 respectively — with the possibility of inviting another user to make his or her own 
decisions — and performing the above-mentioned comparisons. 

The user may also be given the opportunity of copying the current system and inserting 
other point values from another APS that he or she is familiar with — for example, whose 
20 point values ware determined in a different way — and performing the above-mentioned 
comparisons. 

As explained above, linear programming (LP) may be used in the computer program of 
the invention. Specifically, LP may be used at Steps 2 and 4 to determine whether the 
25 resolution of an ambiguity is implied by the inequalities/equalities on the Explicitly 
Resolved list, as well as in Step 1 in determining whether the alternatives on the Loaded 
Alternatives list are ranked and/or the best alternative or alternatives found. In Step 5, LP 
is used for deriving the point values when all ambiguities have been resolved. The 
following features must be observed when LP is used. 

30 
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First, the inequalities/equalities and inherent inequalities must be converted to a form 
suitable for LP. For example, the inherent inequalities a3 > al > al must be written as al 
- al > 1 and a3 - al > 1, and the explicitly resolved ambiguities al + bl = a3 + bl and 
al + b3 > a3 + bl as al + Z>2 - a3 - M = 0 and a2 + 63 - a3 - bl > 1 . Setting the RHS of 
5 the weak inequality to "1" (that is, 'epsilon') corresponds to the initial inequality being 
strict, although other values for epsilon may perform at least as well. 

Second, in Steps 2 and 4, the LP objective function need only be "0", as all that is being 
tested for is the existence of a solution rather than a particular optimal solution. 

10 

Finally, because the variables corresponding to the lowest category on the respective 
criteria (al, 61, cl, and so on) are, effectively, 'numeraires' or 'baseline' values for the 
respective criteria, they can be set equal to zero, thereby eliminating them from the 
system to be solved and increasing the efficiency of the LP algorithm. 

15 

Below is an illustration of how LP may be used to test whether or not an ambiguity on the 
To Be Resolved list is implied by the inequalities/equalities on the Explicitly Resolved list 
by way of example only. 

20 For the sake of the example, suppose the ambiguity in question is a\+ b3 + c3 vs al + bl 
+ cl and the following inequalities/equalities are on the Explicitly Resolved list (where 
the underlined inequality signifies that it is 'redundant' in the sense, as discussed earlier, 
that it is implied by others on the list). 
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a2 + b2 = a3 + bl 
a2 + b3>a3 + b2 
a3+cl>al + c3 
a2 + c\>a\ + c3 
bl+c2 = b3 + cl 
M + c3 = 2»3 + c2 
al+b2 + c3>a2 + bl + cl 



10 

In addition, the inherent inequalities of this exemplar APS with x = 3 and y-3 are: 

a3 > a2 > al 

15 b3>b2>bl 

c3>c2> cl 

These inequalities/equalities (and ignoring the underlined redundant inequality) — and 
given al = M= cl = 0(as discussed above) — are represented in a form suitable for LP, 
20 as follows. By definition, a solution (in terms of the point values) exists to the LP 
problem: 

minimise 0 
subject to: a2 + b2-a3 = 0 

a2 + b3-a3-b2>l 
25 a2-c3>\ 

c2-M = 0 
c3-b3-cl = 0 
b2 + c3-a2^l 

30 a3-o2>l 

b2 > 1 
b3-b2t\ 
c2t\ 
c3-c2>\ 

35 

To test whether al + b3 + c3 vs a2 + bl + cl is implied by the inequalities/equalities on 
the Explicitly Resolved list, the above LP problem is first augmented with b3 + c3 - a2 £ 
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1 (i.e., al + 63 + c3 > a2 + 61 + cl), and tested for whether or not a solution to this new 
problem exists. 

5 

A solution does exist, and so the original LP problem is next augmented with a2 - 63 - 
c3 £ 1 (i.e., o2 + 61 + cl > al + 63 + c3) and tested for whether or not a solution to this 
second new problem exists. 

10 In this case there is no solution, and so it must be inferred that either al + 63 + c3 > a2 + 
61 + cl or al + 63 + c3 = al + M + cl is implied by the inequalities/equalities on the 
Explicitly Resolved list The first inequality above is easily confirmed here by adding b3 
> 62 to the last explicitly resolved ambiguity on the first list above: (63 > 62) + (al + 62 
4- c3 > al + 61 + cl) = (al + 63 + c3 > a2 + 61 + cl). This alternative approach is variant 

15 1 of Approach 1 explained earlier. 

Therefore the ambiguity is implicitly resolved and deleted from the To Be Resolved list. 
(This process is performed for all the ambiguities on the To Be Resolved list.) 

20 In managing the Best Alternatives list in Step 1 above, a similar approach is used to 
determine whether it is possible for one alternative to either be ranked higher ('beat') or 
equal to another. For example, in testing the alternative pair al + bl + c3 vs a2 + 61 + cl, 
the original LP problem is augmented with 62 + c3 - a2 ^ 0. If a solution does not exist, 
then al + 61 + cl is known to be ranked ahead of al + 62 + c3, and al + 62 + c3 is 

25 removed from the Best Alternatives list. Otherwise the original LP problem is next 
augmented with a2 - 62 - c3 £ 0. If a solution does not exist, then al -I- 62 + c3 is known 
to be ranked ahead of a2 + 61 + cl and so a2 + 61 + cl is removed from the Best 
Alternatives list. If a solution did not exist to either the first or the second problems, the 
alternatives are either equal, and so might be ranked top equal, or the relationship 

30 between the alternatives is still ambiguous: and so both alternatives are left on the list. 
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The foregoing describes the invention including preferred forms thereof. Alterations and 
modifications as will be obvious to those skilled in the art are intended to be incorporated 
within the scope hereof, as defined in the accompanying claims. 
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